Large-area imaging by concatenation with array microscope

ABSTRACT

An imaging apparatus consists of multiple miniaturized microscopes arranged into an array capable of simultaneously imaging respective portions of an object. A continuous linear translation approach is followed to scan the object and generate multiple image swaths of the object. In order to improve the quality of the composite image produced by concatenation of the image swaths, the performance of each microscope is normalized to the same base reference for each relevant optical-system property. Correction factors are developed through calibration to equalize the spectral response measured at each detector, to similarly balance the gains and offsets of the detector/light-source combinations associated with the various objectives, to correct for geometric misalignments between microscopes, and to correct optical and chromatic aberrations in each objective.

RELATED APPLICATIONS

This application is a continuation-in-part of U.S. Ser. No. 11/711,283,filed Feb. 27, 2007, which is a continuation-in-part of U.S. Ser. No.10/637,486, filed Aug. 11, 2003, now U.S. Pat. No. 7,184,610, which isbased on PCT/US02/08286, filed Mar. 19, 2002, and claims the benefit ofpriority of U.S. Provisional Application No. 60/276,498, filed Mar. 19,2001, under 35 U.S.C. Section 119. This application is also acontinuation-in-part of U.S. Ser. No. 10,687,432, filed Oct. 16, 2003.

BACKGROUND OF THE INVENTION

1. Field of the Invention

This invention is related in general to the field of microscopy. Inparticular, it relates to a novel approach for acquiring multiple imageswaths of a large sample area using an array microscope and subsequentlycombining the swaths to form a good-quality high-resolution compositeimage.

2. Description of the Related Art

Typical microscope objectives suffer from the inherent limitation ofonly being capable of imaging either a relatively large area with lowresolution or, conversely, a small area with high resolution. Therefore,imaging large areas with high resolution is problematic in conventionalmicroscopy and this limitation has been particularly significant in thefield of biological microscopy, where relatively large samples (in theorder of 20 mm×50 mm, for example) need to be imaged with very highresolution. Multi-element lenses with a large field of view and a highnumerical aperture are available in the field of lithography, but theircost is prohibitive and their use is impractical for biologicalapplications because of the bulk and weight associated with such lenses.

A recent innovation in the field of light microscopy provides a solutionto this problem using a new type of array microscope. As described incommonly owned U.S. Pat. No. 7,184,610, herein incorporated byreference, this new array microscope consists of an array ofminiaturized microscopes wherein each includes a plurality of opticalelements individually positioned with respect to a corresponding imageplane and configured to image respective sections of the sample object.As illustrated in FIG. 1, the array further includes a plurality ofimage sensors corresponding to respective optical elements andconfigured to capture image signals from respective portions of theobject. The absolute magnification in an array microscope is greaterthan one, which means that it is not possible to image the entire objectsurface at once even when it is equal to or smaller than the size of thearray. Rather, the imaged portions of the object are necessarilyinterspaced in checkerboard fashion with parts of the object that arenot imaged. Accordingly, this array microscope was designed inconjunction with the concept of linear object scanning, where the objectis moved relative to the array microscope and data are acquiredcontinuously from a collection of linear detectors. Data swaths obtainedfrom individual optical systems are then concatenated to form thecomposite image of the object.

In such an array microscope, a linear array of miniaturized microscopesis preferably provided with adjacent fields of view that span across afirst dimension of the object and the object is translated past thefields of view across a second dimension to image the entire object. Asillustrated in FIG. 2, because each miniaturized microscope 10 is largerthan its field of view 12 (having, for example, respective diameters ofabout 1.8 mm and 200 μm—only one is shown for simplicity), theindividual microscopes of the imaging array are staggered in thedirection of scanning (x direction in the figure) so that theirrelatively smaller fields of view are offset over the second dimensionbut aligned over the first dimension. As a result of such staggeredarrangement of the rows of miniaturized microscopes, the continuousswaths covered by the linear scan of each optical system issubstantially free of overlap with the continuous swaths covered byadjacent optical systems. At each acquisition frame each miniaturizedmicroscope projects image data for a small section of the sample objectdirectly onto a detector and the individual frame data are then used toform an image of the entire sample object by hardware or softwaremanipulation.

The axial position of the array with respect to the sample object ispreferably adjusted to ensure that all parts of the sample surface areimaged in a best-focus position. Thus, the detector array provides aneffectively continuous coverage along the first dimension whicheliminates the need for mechanical translation of the microscope in thatdirection, providing a highly advantageous increase in imaging speed bypermitting complete coverage of the sample surface with a singlescanning pass along the second dimension. Such miniaturized microscopesare capable of imaging with very high resolution. Therefore, large areasare imaged without size limitation and with the very high resolutionafforded by the miniaturized microscopes.

In a similar effort to provide a solution to the challenge of imaginglarge areas with high magnification, U.S. Pat. No. 6,320,174 (Tafas etal.) describes a system wherein an array of optical elements is used toacquire multiple sets of checkerboard images that are then combined toform a composite image of the sample surface. The sample stage is movedin stepwise fashion in relation to the array of microscopes (so called“step-and-repeat” mode of acquisition) and the position of the samplecorresponding to each data-acquisition frame is recorded. The variousimage tiles are then combined in some fashion to provide the object'simage. The patent does not provide any teaching regarding the way suchmultiple sets of checkerboard images may be combined to produce ahigh-quality high-resolution composite image. In fact, while stitchingtechniques are well known and used routinely to successfully combineindividual image tiles, the combination of checkerboard images presentsnovel and unique problems that cannot be solved simply by theapplication of known stitching techniques.

For example, physical differences in the structures of individualminiaturized objectives and tolerances in the precision with which thearray of microscopes is assembled necessarily produce misalignments withrespect to a common coordinate reference. Moreover, optical aberrationsand especially distortion and chromatic aberrations, as well as spectralresponse and gain/offset properties, are certain to vary from microscopeto microscope, thereby producing a checkerboard of images of non-uniformquality and characteristics. Therefore, the subsequent stitching byconventional means of multiple checkerboards of image tiles acquiredduring a scan cannot produce a high-resolution composite image thatprecisely and seamlessly represents the sample surface.

For instance, as illustrated in FIG. 3, assume that a conventionalstep-and-repeat array microscope of the type described by Tafas et al.includes only two adjacent miniaturized microscopes producing respectiveimages 14 and 16 and that the second microscope introduces a slightrotation and offset in the image 16 acquired from the sample surface 18with respect to the image 14 acquired by the first microscope (thedashed line represents a perfectly aligned image). In such case theacquisition of the first frame of image tiles 14,16 would produce apattern similar to that illustrated in the figure. The acquisition ofthe second frame of image tiles 14′ and 16′ would produce a similarlymisaligned set of images, as illustrated.

If conventional stitching procedures are used to combine the variousimage tiles so acquired, such as described in U.S. Pat. No. 5,991,461and No. 6,185,315, the stitching of images 14 and 14′ will produce aseamless image of uniform quality accurately representing thecorresponding section of the sample surface 12. This is because bothimages 14 and 14′ result from data acquired with the same miniaturizedmicroscope and the same spectral response, gain, offset, distortion andchromatic aberrations (to the extent they have not been removed bycorrection) apply to both images, thereby producing a composite image ofuniform quality. Inasmuch as stitching procedures exist that are capableof correcting misalignments between adjacent image tiles, a similarresult could be obtained by stitching images 16 and 16′.

The combination of images acquired with different microscopes, however,could not be carried out meaningfully with conventional stitchingtechniques. Combining image 14′ with image 16, for example, may bepossible as far as misalignments and offsets are concerned, but thecombined difference could still be non-uniform with respect to spectralresponse, gain, offset, and distortion or chromatic aberrations(depending on the characteristics of each miniaturized microscope).Therefore, the overall composite image could represent a meaninglessassembly of incompatible image tiles that are incapable of producing anintegrated result (like combining apples and oranges).

Thus, the prior art does not provide a practical approach to the verydesirable objective of imaging a large area with an array microscope insequential steps to produce checkerboards of images that can later becombined in a single operation simply by aligning any pair of adjacentimage tiles. Similarly, the prior art does not provide a solution to thesame problem of image non-uniformity produced by the array microscope ofU.S. Pat. No. 7,184,610 that is scanned linearly over a large area ofthe sample surface to produce image swaths that are later combined toform a composite image.

U.S. Pat. No. 5,768,443 (Michael et al.) describes a method forcoordinating the fields of view of a multi-camera machine bypre-calibrating each camera for distortion to develop adistortion-correction map, applying the correction to images acquiredsimultaneously with each camera, and combining the corrected images toproduce a composite image. While the Michel patent does not relate tomicroscopes, it provides a useful approach to solving the problemaddressed by the present invention. This invention provides a generaland efficient solution toward this end using the linear-scan arraymicroscope of U.S. Pat. No. 7,184,610.

BRIEF SUMMARY OF THE INVENTION

In view of the foregoing, the invention is described for simplicity withreference to an array microscope operating in step-and-repeat scanningmode, but it is applicable to every situation where an array microscopeis used to generate images of portions of a large sample area to besubsequently combined to image the whole area. In fact, the preferredimaging apparatus consists of multiple optical systems arranged into anarray capable of simultaneously imaging a portion of an object with asingle linear scan in the manner described in U.S. Pat. No. 7,184,610.

In order to enable the combination of the various multi-image framesacquired during a scan in a seamless manner to compose a large-areaimage with uniform and significant features, the performance of eachmicroscope is normalized to the same reference base for each relevantoptical-system property. Specifically, a correction-factor matrix isdeveloped through calibration to equalize the spectral response measuredat each detector and preferably also to similarly balance the gains andoffsets of the detector/light-sources associated with the variousobjectives, to correct for geometric misalignments between microscopes,and to correct distortion, chromatic, and other aberrations in eachobjective.

Thus, by applying the resulting correction-factor matrix to the dataacquired by scanning the sample object in any fashion, the resultingcheckerboard images are normalized to a uniform basis so that they canbe concatenated or combined by stitching without further processing. Asa result of this normalization process, the concatenation or stitchingoperation can be advantageously performed rapidly and accurately for theentire composite image simply by aligning pairs of adjacent images fromthe image checkerboards acquired during the scan. A single pair ofimages from each pair of checkerboards is sufficient because theremaining images are automatically aligned as well to produce a uniformresult by virtue of their fixed spatial position within thecheckerboard.

Various other purposes and advantages of the invention will become clearfrom its description in the specification that follows and from thenovel features particularly pointed out in the appended claims.Therefore, to the accomplishment of the objectives described above, thisinvention consists of the features hereinafter illustrated in thedrawings, fully described in the detailed description of the preferredembodiment and particularly pointed out in the claims. However, suchdrawings and description disclose but one of the various ways in whichthe invention may be practiced.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates the basic configuration of an array microscopecomposed of several individual optical elements formed on rectangulargrids aligned along common optical axes.

FIG. 2 is a simplified schematic representation of the fields of view inan array-microscope layout of 99 miniaturized microscopes in an array of9 rows of individual microscopes disposed transversely and 11 columns ofmicroscopes disposed at a slight angle with respect to the direction ofscanning, such that a complete coverage of the sample surface isachieved with a single linear scan.

FIG. 3 is a simplified schematic representation of the images producedby a two-microscope array at two acquisition frames taken after thesample is moved by a predetermined amount designed to cover a samplearea larger than the field of view of the array microscope. The figureillustrates a physical misalignment in the image produced by the secondmicroscope with respect to the image produced by the first microscope.

FIG. 4 illustrates a large area of an object surface covered by thefield of view of an objective scanning in overlapping fashion through astep-and-repeat process.

FIG. 5 illustrates the large area of FIG. 4 covered by the fields ofview of individual objectives of a four-element array microscope at aninitial scanning position.

FIG. 6 is the image produced by scanning a sample area with an arraymicroscope in step-and-repeat mode without correction for distortion.

FIG. 7 is an enlarged view of a section in the overlap region betweenadjacent tiles that clearly illustrates the distortion error of FIG. 6,as exhibited by the vertical discontinuities in the data.

FIG. 8 is the image produced by scanning the same sample of FIG. 6without correction for geometric misalignments between the twomicroscopes.

FIG. 9 is the image produced with the same sample data after calibrationof the array microscope for geometric uniformity, as described, and thesubsequent application of the resulting correction factors to the rawimage data.

FIG. 10 is an image produced by scanning the same sample of FIG. 6without correction for spectral-response uniformity between the twomicroscopes.

FIG. 11 is the image produced with the same data of FIG. 8 aftercalibration of the array for spectral-response uniformity according tothe invention and the subsequent application of the resulting correctionfactors to the raw image data.

FIG. 12 is an image produced by scanning a sample area with an arraymicroscope in step-and-repeat mode without correction for differences indetector gain and offset.

FIG. 13 is an image produced with the same data of FIG. 12 aftercalibration of the array for gain and offset uniformity according to theinvention and the subsequent application of the resulting correctioncoefficients to the raw image data.

FIG. 14 shows a composite image produced by concatenating image swathsproduced with linear scanning without correction.

FIG. 15 shows the composite image produced by the same data afterapplication of the correction matrix according to the invention.

FIG. 16 illustrates a current embodiment of the linear scanning arraymicroscope preferred to practice the invention.

DESCRIPTION OF THE PREFERRED EMBODIMENTS OF THE INVENTION

The invention was motivated by the realization that the images producedby data acquisition using an array microscope cannot be combineddirectly to produce a uniform composite image because of the unavoidabledata incompatibilities produced by discrepancies in the opticalproperties of the various miniaturized microscopes in the array. Theheart of the invention lies in the idea of normalizing such opticalproperties to a common basis, so that functionally the array ofmicroscopes performs, can be viewed, and can be treated as a singleoptical device of uniform characteristics. As a result, each set ofmultiple images produced simultaneously at each scanning step can beviewed and treated as a single image that can be aligned and combined inconventional manner with other sets in a single operation to produce thecomposite image of a large area. The advantages of the invention areparticularly useful when the array microscope is utilized with linearscanning and image swaths are concatenated or stitched together.

As used herein, the term “checkerboard” refers, both in relation tostep-and-repeat scanning and linear scanning, to image framescorresponding to portions of the sample object interspaced incheckerboard fashion with parts of the object that are not imaged. Theterm “microscope” is used with reference to both the array microscopeand the individual miniaturized microscopes within the array, and it isassumed that the distinction will be apparent to those skilled in theart from the context of the description. The term “field of view” issimilarly applied to both. The term “axial” is intended to refer to thedirection of the optical axis of the array microscope used for theinvention. “Step-and-repeat” is used to refer to a data acquisition modewherein frames of data are taken during a scan by translating the objector the microscope in stepwise fashion and by acquiring data staticallybetween steps. With reference to data acquisition, the term “frame” isused to refer to the simultaneously acquired data obtained at any timewhen the system's sensor or sensors operate to acquire data. The term“tile” refers both to the portion of the sample surface imaged by asingle miniaturized microscope in an array and to the image so produced.The terms “swath” and “image swath” refer to the image produced by eachindividual objective in an array microscope of the type described inU.S. Pat. No. 7,184,610 as a result of a single linear scan, each swathproducing the image of a corresponding strip of object surface equal inwidth to the field of view of the individual objective and in length tothe span of the linear scan.

The term “concatenation” refers to the process of joining the tile orswath images of adjacent portions of the sample surface acquired withoutfield-of-view overlaps, based only on knowledge of the spatial positionof each objective in relation to a reference system and the assumptionthat such knowledge is correct. In contrast to concatenation, the term“stitching” refers to any conventional procedure used to combineseparate data sets corresponding to adjacent sections of a samplesurface acquired with field-of-view overlaps, wherein the knowledge ofthe spatial position of each objective is used only as an approximationand such overlaps are used to precisely align the images to form aseamless composite.

Moreover, the terms “geometric alignment,” “geometric calibration” and“geometric correction” are used herein with reference to linear (in x, yor z) and/or angular alignment between image tiles produced by an arraymicroscope (distortion), and to chromatic aberration associated with themicroscopes in the array. The term “spectral response” refers to thesignals registered by the detectors in response to the light receivedfrom the imaging process. Finally, the terms “gain” and “offset”variations are used with reference to differences in the electricalresponse measured at different pixels or on average at differentdetectors as a function of variations in the current supplied to thelight sources, in the background light received by each microscope, inthe properties of the optical systems, in the detector pixel responses,in the temperature of the sensors, and in any other factor that mayaffect gain and offset in an optical/electronic system.

Referring to the drawings, wherein like reference numerals and symbolsare used throughout to designate like parts, FIG. 1 illustrates thebasic configuration of an array microscope 20 as described in U.S. Pat.No. 7,184,610. The array is composed of several individual opticalelements (22 a,24 a,26 a) formed on rectangular plates 22,24,26 (nineelements are illustrated in each grid in the figure). Each microscopeincludes three lenses distributed along an optical axis passing throughthe plates 22, 24 and 26. These plates are arranged into a stack 28aligned with another plate 30 containing area detectors 30 a positionedin the image plane of the object surface 18 to be measured. Each areadetector 30 a consists of many individual sensing elements (pixels) thatare preferably implemented with charged-coupled-device (CCD) orcomplementary-metal-oxide-semiconductor (CMOS) arrays, or any othersuitable image sensing device.

As illustrated, this embodiment is characterized by a one-to-onecorrespondence between each optical system and an area detector. Thus,the field of view of each detector projected through the optical systemyields a rectangular image (as shaped by the detector) received in theimage plane of the object. At each given acquisition time, theindividual optical systems (objectives) of the array form the image of aportion of the object surface on the corresponding detector. Theseimages are then read out by suitable electronic circuitry (not shown)either simultaneously or in series.

FIG. 4 illustrates, using step-and-repeat scanning for convenience ofillustration, a strategy that can be used advantageously to acquirehigh-resolution images of an object area 32 larger than the field ofview covered by the microscope objectives. As mentioned above, whengreater-than-one absolute magnification is involved, it is not possibleto image the entire object surface at once because of the physicalconstraints imposed by the size of the objectives. Therefore, it isnecessary to repeat the acquisition of data at locations of the objectthat have not been previously imaged. The various tiles 34 of data arethen combined in a concatenating or a stitching operation that requiresa large number of images to be taken and stored. When doneconventionally, the process takes significant time because for each dataframe the object must be positioned in front of the optical system andoften also focused before acquisition.

Such repetitive procedure is not practical (or sometimes even possible)with conventional microscopes because of their size. The typical fieldof view of a 40H microscopic objective is about 300 im with a lens about20 mm in diameter. Therefore, even an array of conventional microscopes(such as described in U.S. Pat. No. 6,320,174) could not image more thana single tile at a time on an object surface of about 20×20 mm or lessin size. By comparison, the field of view of each individual opticalsystem in an array of miniaturized microscopes is comparable in size(i.e., about 200 μm), but the distance between optical systems can be assmall as 1.5 mm. Thus, the diameter to field-of-view ratio in arraymicroscopes is about 7.5 while in conventional optical microscopes it isin the order of about 65. As a result, array microscopes are mostsuitable for acquiring simultaneously multiple images of portions of thesample object in checkerboard fashion. Because imaging by the variousminiaturized objectives is performed in parallel, multiple tiles areimaged at the same time at each scanning step. This can be done bytranslating the array microscope with respect to the object (or viceversa) on some predetermined step pattern.

For example, the areas 36 shaded in gray in FIG. 5 would correspond toimage tiles of corresponding portions of the object surface 32 assumingthat the array microscope had four optical systems with fields of viewspaced apart on the object plane by a distance substantially equal tothe field of view of each individual microscope (in this example adistance to field-of-view ratio of about 2 is assumed to simplify theillustration). In fact, a smaller distance is often required in practiceto allow sufficient overlaps (the cross-hatched areas 38, magnified inthe figure for clarity of illustration) during the acquisition steps toaccount for scanning misalignments and/or to acquire duplicate data thatfacilitate the subsequent stitching of adjacent tiles. While a typicalstep-and-repeat procedure with a single microscope requires as manyacquisition frames as the number of tiles 34 necessary to cover theoverall imaged area 32, the use of an array microscope makes it possibleto reduce the frames by a factor equal to the number of objectivesaggregated in the array. In the example shown in FIGS. 4 and 5, a totalof sixteen individual tiles would be necessary to reconstruct the entireobject surface. Thus, using a conventional microscope, sixteenacquisitions would be required. Using an array microscope with fouroptical systems, it is instead possible to acquire four tilessimultaneously at each step and the entire surface can be imaged in justfour steps. In the first step, the images corresponding to fields ofview S11, S13, S31 and S33 are acquired; followed by S21, S41, S23 andS43; then S22, S42, S24 and S44; and finally S12, S32, S14 and S34. Thefour checkerboard frames of images so acquired are then concatenated orstitched together to form the composite image of the object surface.

As described, in order to enable the composition of a seamless,meaningful image of the large area for which data have been acquired,the system is calibrated according to the invention and the results ofcalibration are applied to the frames of data prior to stitching orconcatenation. According to one aspect of the invention, the device isfirst calibrated to establish the relative position and themagnification (pixel spacing) of each field of view at imaging colorbands (RGB) and corrective factors are then applied to align all imagetiles (with respect to a fixed coordinate system) and to produce uniformmagnification across the array microscope. That is, the system iscorrected for aberrations commonly referred to in the art as distortionand chromatic aberration. Such calibration may be accomplished, forexample, using prior knowledge about the geometry of the system or usingstandard correlation methods. In the former case, each tile's image isreconstructed, if necessary, according to such prior knowledge byapplying geometric transformations (such as rotation, scaling, and/orcompensation for distortion) designed to correct physicalnon-uniformities between objectives and optical aberrations within eachobjective. The images are then concatenated or stitched to create acomposite image.

Because of the simultaneous acquisition of each checkerboard set ofimages (S11, S13, S31 and S33, for example), the geometric relationshipbetween individual optical systems in the array is preserved betweenacquisition frames. Therefore, this fixed relationship can be usedadvantageously to materially speed up the image combination process.Since the physical relationship between checkerboard images does notchange between frames, once normalized according to the invention, thesequence of frames can be concatenated or stitched directly withoutfurther processing subject only to alignment to correct scanningpositioning errors. Thus, using conventional stitching methods toseamlessly join two adjacent tile images acquired in consecutive steps(S11 and S21, for example), the rest of the tile images (S13,S31,S33 andS23,S41,S43) can be placed directly in the composite image simply byretaining their relative positions with respect to S11 and S21,respectively.

As part of the calibration procedure, this relationship can beestablished by imaging a reference surface or target through which theposition and orientation of each field of view can be uniquely andaccurately identified. One such reference surface could be, for example,a flat glass slide with a pattern of precisely positioned crosses on arectangular grid that includes a linear ruler with an accurate scale.Such a reference target can be easily produced using conventionallithography processes with an accuracy of 0.1 μm or better. Using alarge number of individual target points for the calibration procedurecan further increase the accuracy.

The lateral position, angular orientation, and distortion of eachoptical system and detector can be accurately measured by determiningthe positions of reference marks (such as points on the crosses) withinthe field of view of each image and by comparing that information withthe corresponding positions of those marks in the reference surfacebased on the ruler imprinted on it. The differences are converted inconventional manner to correction factors that can then be used tocorrect image errors due to the geometric characteristics of the arraymicroscope. As a result, linear and angular misalignment of the variousfields of view in the array can be corrected to establish the exactposition of each tile within the overall composite image. Once soestablished, such correction factors can be incorporated in firmware toincrease the processing speed of the optical system.

Alternatively, correlation methods can be used that rely only on anapproximate knowledge about the position of each individual image in thecheckerboard of fields of view. Using these techniques, the exactposition of each tile is established by matching two overlappingsections of images of adjacent portions of the object (taken atdifferent frames). This can be done in known manner using, for instance,a maximum cross-correlation algorithm such as described by Wyant andSchmit in “Large Field of View, High Spatial Resolution, SurfaceMeasurements,” Int. J. Mach. Tools Manufact. 38, Nos. 5-6, pp. 691-698(1998). Thus, this approach requires an overlap between adjacent fieldsof view, as illustrated in FIGS. 4 and 5. Typical overlaps are on theorder of 10% of the tile size.

It is noted that typical optical systems used in imaging produce aninverted image; that is, the orientation of the x and y axes of theobject are opposite in sample surface and in the image. Therefore, intheir raw form these images cannot be used to construct a compositeimage. Rather, before either concatenation or stitching of the varioustiles is carried out, each image needs to be inverted to match theorientation of the object. This operation can be done in conventionalmanner either in software or hardware.

FIGS. 6, 7, 8 and 9 illustrate the positive results produced by thegeometric-calibration procedure of the invention. FIG. 6 shows the imageproduced by scanning a sample area with a two-microscope array (theexample is limited to two tiles for simplicity) in step-and-repeat mode(only two frames are illustrated) without correction for distortion. Thedistortion error introduced by the right-hand microscope is illustratedin dashed line next to the ideal distortion-free fields of viewcharacterized by the overlapping solid-line rectangles. The distortionis more clearly visible in the enlarged view of FIG. 7, whichcorresponds to a section of the overlap region wherein the distortionerror produces vertical discontinuities in the data. FIG. 8 is the imageproduced by scanning the same sample without correction for geometricmisalignments between the two microscopes. The angular error introducedby the right-hand microscope is illustrated by the corresponding tiltedfields of view. FIG. 9 is the image produced with the same sample dataafter calibration of the array microscope for geometric uniformity, asdescribed, and the subsequent application of the resulting correctionfactors to the raw image data. In all cases the checkerboard imagescorresponding to each acquisition frame were aligned and stitchedtogether applying the maximum cross-correlation algorithm to a singlepair of adjacent image tiles.

According to another aspect of the invention, the device is calibratedto establish a uniform spectral response across the array microscope.That is, correction factors are generated to normalize the spectralresponse of each detector in the array. When images belonging todifferent fields of view are acquired using separate detectors and/orlight sources, there is a possibility of variation in the spectralresponses obtained from the various detectors. These differences maystem, for example, from light sources illuminating different fields ofview at different temperatures, or from different ages of the lightbulbs, or from different filter characteristics, etc. These differencesalso need to be addressed and normalized in order to produce a compositeimage of uniform quality, especially when the images are subject tosubsequent computer analysis, such as described in U.S. Pat. No.6,404,916. Similar differences may be present in the spectral responseof the detectors as a result of variations in the manufacturing processor coating properties of the various detectors.

A suitable calibration procedure for spectral response to establishcorrection factors for each field of view may be performed, for example,by measuring the response to a set of predefined target signals, such ascalibrated illumination through color filters. For each field of viewthe response to red, green and blue channels can be calculated using anyone of several prior-art methods, such as described in W. Gross et al.,“Correctability and Long-Term Stability of Infrared Focal Plane Arrays”,Optical Engineering, Vol. 38(5), pp. 862-869, May 1999; and in A.Fiedenberg et al., “Nonuniformity Two-Point Linear Correction Errors inInfrared Focal Plane Arrays,” Optical Engineering, Vol. 37(4), pp.1251-1253, April 1998. The images acquired from the system can then becorrected for any non-uniformity across an individual field of view oracross the entire array. As one skilled in the art would readilyunderstand, correction factors may be implemented in the form of look-uptables or correction curves applied to the acquired images. Thecorrection for differences in the spectral response can be carried outon the fly through computation during data acquisition, such as by usinga programmable hardware device. Alternatively, the correction may beimplemented structurally by modifying the light-source/detector opticalpath to produce the required compensation (for example, by insertingcorrection filters, changing the temperature of the light source, etc.).

It is understood that all these procedures aim at producing a uniformspectral response in each acquisition system, such that no variation inthe image characteristics is produced as a result of devicenon-uniformities across the entire composite image. Therefore, in casewhen such a corrective procedure is not carried out prior to theformation of the composite image, the spectral-response characteristicsof each field of view should be used in post-imaging analysis tocompensate for differences. As one skilled in the art would readilyunderstand, these corrections can also be applied to a detector as awhole or on a pixel-by-pixel basis.

FIGS. 10 and 11 illustrate the positive results produced by thespectral-response calibration procedure of the invention. FIG. 10 is animage produced by scanning a sample area with a two-microscope array instep-and-repeat mode without correction for differences in spectralresponse. The figure shows that the left and right microscopes exhibit arelatively higher spectral response to red and green, respectively(which produce darker and lighter pictures, respectively, in black andwhite). FIG. 11 is the corresponding image produced with the same dataafter calibration of the array for spectral-response uniformity, asdescribed, and the subsequent application of the resulting correctionfactors to the raw image data. The correction factors were calculatedfor each pixel of each field of view in response to red, green and bluechannels using the methods described in W. Gross et al. and A.Fiedenberg et al., supra.

According to yet another aspect of the invention, the device iscalibrated to establish a uniform gain and offset response across thearray microscope. Because of variations in the currents supplied to thelight sources to the various optical systems, in the optical propertiesof the systems, in the detector pixel responses, in the temperatures ofthe sensors, etc., the combined response of the instrument to light mayvary from pixel to pixel and from one field of view to another. Suchvariations are manifested in the composite image as sections withdifferent properties (such as brightness and contrast). In addition,such non-uniform images may cause different responses to each field ofview to be obtained when automated analysis tools are used. Therefore,it is important that these variations also be accounted for bycalibration, which can be achieved by measuring the response produced bya know target to generate a set of gain and offset coefficients. Forexample, a known target is placed in the field of view of each opticalsystem (such a target could be a neutral-density filter with knownoptical density). A series of images is taken for different opticaldensity values. Based on these measurements, the gain and offset of eachpixel in each field of view are calculated using one of several wellknown procedures, such as outlined in W. Gross et al. and A. Fiedenberget al., supra. Appropriate correction coefficients are then computed tonormalize the image properties of each pixel (or on average for a fieldof view) so that the same gain/offset response is measured across theentire set of fields of view. A single target gain and a single targetoffset may be used to normalize the response of thedetector/light-source combination at two signal levels and, assuminglinear behavior, the resulting correction factors may be used betweenthose levels. Correction factors for additional linear segments ofsignal levels may be similarly computed, if necessary, to cover agreater signal intensity span.

FIGS. 12 and 13 illustrate the improvements produced by the gain/offsetcalibration procedure of the invention. FIG. 12 is an image produced byscanning a uniform-background sample area with the same array microscopein step-and-repeat mode without correction for differences in detectorgain and offset. The difference in the intensity registered by adjacentlines of pixels clearly illustrates non-uniformity in the response ofthe various detector pixels. FIG. 13 is the corresponding image producedwith the same data after calibration of the array for gain and offsetuniformity according to the invention, as described, and the subsequentapplication of the resulting correction coefficients to the raw imagedata. The correction coefficients were calculated for each pixel of eachfield of view using the procedure described in W. Gross et al. and A.Fiedenberg et al., supra.

Obviously, all three kinds of corrections described herein may be, andpreferably are, implemented at the same time on the image data acquiredto produce a composite image. To the extent that normalizing correctionsare implemented through linear transformations, a cumulative matrix ofcoefficients can be calculated and used to effect one, two or all threekinds of corrections. In addition, as mentioned above, a composite imagecan be constructed using either a concatenation or a stitchingtechnique. The first method is preferred because of its speed, but it isalso much more difficult to implement in step-and-repeat mode becausethe exact position of each tile in the patchwork of images (orcheckerboards) acquired in successive frames with an array microscopeneeds to be known with an accuracy better than the sampling distance.Thus, in order to improve the knowledge about the relative position ofeach field of view at each frame, the image acquisition is preferablycarried out at the same instant for all detectors in the arraymicroscope device. This requires a means of synchronization of alldetectors in the system. One approach is to use one of the detectors asa master and the rest of detectors as slaves. Another approach is to usean external synchronization signal, such as one coupled to a positionsensor for the stage, or a signal produced by stroboscopic illumination,or one synchronized to the light source.

Alternatively, a less precise knowledge about the position of each fieldof view can be combined with conventional stitching techniques toconstruction of the composite image. Each checkerboard of imagesacquired simultaneously at each frame can be used as a single imagebecause the geometric relationship between the images is preservedduring the stitching process. Thus, each checkerboard frame isseamlessly fused with adjacent ones in the composite image simply byapplying conventional stitching (such as correlation techniques) to asingle pair of adjacent images, knowing that the remaining images remainin fixed relation to them. Such technique can significantly speed up theprocess of overall image construction when the exact position of eachcheckerboard in the composite image is not known.

The procedure herein disclosed gives good results for objects that areflat within the depth of field of the individual optical systems. Forobjects that extend beyond such depth of field, additional refocusingmay be required. This can be done most conveniently using an arraymicroscope where each of the optical systems can be focusedindependently. Another way to compensate for variations in object heightis to use a cubic phase plate, as described in U.S. Pat. No. 6,069,738.

It is noted that the invention has been described for convenience interms of an array microscope adapted to scan in step-and-repeat fashion,but the invention is particularly advantageous when applied to an arraymicroscope configured to scan linearly entire the length of the objectsurface, as described in U.S. Pat. No. 7,184,610 and illustrated in FIG.16. In such preferred case, the linear array of miniaturized microscopesis provided with adjacent fields of view that span across a firstdimension of the object (as illustrated in FIG. 2), and the object istranslated past the fields of view across a second dimension to imagethe entire object. As described above, because each miniaturizedmicroscope is larger than its field of view, the individual microscopesof the imaging array are staggered in the direction of scanning so thattheir relatively smaller fields of view are offset over the seconddimension but aligned over the first dimension. Thus, the detector arrayprovides an effectively continuous linear coverage along the firstdimension, which eliminates the need for mechanical translation of themicroscope in that direction and provides a highly advantageous increasein imaging speed by permitting complete coverage of the sample surfacewith a single scanning pass along the second dimension. However,inasmuch as a composite picture is created by combining swaths measuredby individual microscopes and associated detectors, the same criticalneed exists for uniformity in the characteristics of the images acquiredacross the array.

The great advantage afforded by the microscope array of U.S. Pat. No.7,184,610 lies in the fact that a single linear scan produces aplurality of image swaths having a width defined by the field of view ofeach microscope in the array and a length defined by the length of thescan across the object. As a result of this configuration, the imageproduced by the concatenation of this plurality of image swaths obtainedin a single scan extends over the width of the microscope array and thelength of the scan over the object. If the objectives are appropriatelystaggered, each image swath is precisely aligned with the two swathsadjacent to it, so that concatenation can be carried out rapidly withoutfurther processing.

FIGS. 14 and 15 illustrate the invention as applied to a linear scanningarray wherein a correction matrix is used to account for gain and offsetuniformity. FIG. 14 shows a composite image produced by concatenatingimage swaths without correction. FIG. 15 shows the composite imageproduced by the same data after application of the correction matrixaccording to the invention.

Thus, a method has been disclosed to produce a seamless color compositeimage of a large object area by acquiring data in with an arraymicroscope. The method teaches the concept of normalizing all individualmicroscopes to produce images corrected for spatial misalignments andhaving uniform spectral-response, gain, offset, and aberrationcharacteristics.

Therefore, while the invention has been shown and described herein inwhat is believed to be the most practical and preferred embodiments withreference to array microscopes operating in step-and-repeat scanningmode, it is recognized that it is similarly applicable to linearscanning. Accordingly, it is further understood that departures can bemade within the scope of the invention, which is not to be limited tothe details disclosed herein but is to be accorded the full scope of theclaims so as to embrace any and all equivalent methods and products.

1. A method of combining multiple swaths of images acquired with alinear scan of an object with an array microscope, comprising thefollowing steps: providing a two-dimensional microscope array with aplurality of magnifying imaging systems disposed along a correspondingplurality of optical axes for imaging a picture of the object onto adetector, said plurality of magnifying imaging systems being arranged inrows and configured to image respective sections of the object, whereinsaid rows of imaging systems are staggered with respect to a lineardirection of scan across the object, such that each of the imagingsystems acquires image data corresponding to a respective continuousstrip of the object along said linear direction of scan; providing ascanning mechanism for producing a relative movement between themicroscope array and the object, said scanning mechanism operatingcontinuously along said linear direction of scan across the object;calibrating said array microscope to derive correction factors fordistortion in said image data; scanning the object to produce multipleadjacent image swaths of the object, each swath having a width definedby a field of view of a corresponding magnifying imaging system in thearray and a length defined by said linear direction of scan across theobject; applying said correction factors to said multiple image swathsto obtain multiple swaths of corrected images; and concatenating saidmultiple swaths of corrected images to produce a composite image of theobject.
 2. The method of claim 1, wherein said calibrating step furtherincludes deriving correction factors for chromatic aberrations producedby the array microscope in said image data.
 3. The method of claim 1,wherein said calibrating step further includes deriving correctionfactors for producing a uniform spectral response throughout the arraymicroscope.
 4. The method of claim 1, wherein said calibrating stepfurther includes deriving correction factors for producing a uniformgain throughout the array microscope.
 5. The method of claim 1, whereinsaid calibrating step further includes deriving correction factors forproducing a uniform offset throughout the array microscope.
 6. A methodof combining multiple image swaths acquired in a linear scan of anobject with an array microscope, comprising the following steps:providing a two-dimensional microscope array with a plurality ofmagnifying imaging systems disposed along a corresponding plurality ofoptical axes for imaging a picture of the object onto a detector, saidplurality of magnifying imaging systems being arranged in rows andconfigured to image respective sections of the object, wherein said rowsof imaging systems are staggered with respect to a linear direction ofscan across the object, such that each of the imaging systems acquiresimage data corresponding to a respective continuous strip of the objectalong said linear direction of scan; providing a scanning mechanism forproducing a relative movement between the microscope array and theobject, said scanning mechanism operating continuously along said lineardirection of scan across the object; calibrating said array microscopeto derive correction factors to produce a uniform spectral responsethroughout said array microscope; scanning the object to producemultiple adjacent image swaths of the object, each swath having a widthdefined by a field of view of a corresponding magnifying imaging systemin the array and a length defined by said linear direction of scanacross the object; applying said correction factors to said multipleimage swaths to obtain multiple swaths of corrected images; andconcatenating said multiple swaths of corrected images to produce acomposite image of the object.
 7. The method of claim 6, wherein saidcalibrating step further includes deriving correction factors forchromatic aberrations produced by the array microscope in said imagedata.
 8. The method of claim 6, wherein said calibrating step furtherincludes deriving correction factors for producing a uniform gainthroughout the array microscope.
 9. The method of claim 6, wherein saidcalibrating step further includes deriving correction factors forproducing a uniform offset throughout the array microscope.
 10. A methodof combining multiple image swaths acquired in a linear scan of anobject with an array microscope, comprising the following steps:providing a two-dimensional microscope array with a plurality ofmagnifying imaging systems disposed along a corresponding plurality ofoptical axes for imaging a picture of the object onto a detector, saidplurality of magnifying imaging systems being arranged in rows andconfigured to image respective sections of the object, wherein said rowsof imaging systems are staggered with respect to a linear direction ofscan across the object, such that each of the imaging systems acquiresimage data corresponding to a respective continuous strip of the objectalong said linear direction of scan; providing a scanning mechanism forproducing a relative movement between the microscope array and theobject, said scanning mechanism operating continuously along said lineardirection of scan across the object; calibrating said array microscopeto derive correction factors to produce a uniform gain throughout saidarray microscope; scanning the object to produce multiple adjacent imageswaths of the object, each swath having a width defined by a field ofview of a corresponding magnifying imaging system in the array and alength defined by said linear direction of scan across the object;applying said correction factors to said multiple image swaths to obtainmultiple swaths of corrected images; and concatenating said multipleswaths of corrected images to produce a composite image of the object.11. The method of claim 10, wherein said calibrating step furtherincludes deriving correction factors for producing a uniform offsetthroughout the array microscope.
 12. The method of claim 10, whereinsaid calibrating step further includes deriving correction factors forchromatic aberrations produced by the array microscope in said images.13. The method of claim 10, wherein said calibrating step furtherincludes deriving correction factors for producing a uniform spectralresponse throughout the array microscope.
 14. A method of imaging anobject with an array microscope comprising the following steps:providing a two-dimensional microscope array with a plurality ofmagnifying imaging systems disposed along a corresponding plurality ofoptical axes for imaging a picture of the object onto a detector, saidplurality of magnifying imaging systems being arranged in rows andconfigured to image respective sections of the object, wherein said rowsof imaging systems are staggered with respect to a linear direction ofscan across the object, such that each of the imaging systems acquiresimage data corresponding to a respective continuous strip of the objectalong said linear direction of scan; providing a scanning mechanism forproducing a relative movement between the microscope array and theobject, said scanning mechanism operating continuously along said lineardirection of scan across the object; calibrating the array microscope toderive correction factors designed to correct imaging characteristics ofsaid magnifying imaging systems in the array microscope in order tonormalize an output thereof and produce images with uniform opticalproperties; scanning the object to produce multiple adjacent imageswaths of the object, each swath having a width defined by a field ofview of a corresponding magnifying imaging system in the array and alength defined by said linear direction of scan across the object;applying said correction factors to said multiple image swaths to obtainmultiple swaths of corrected images; and concatenating said multipleswaths of corrected images to produce a composite image of the object.15. The method of claim 14, wherein said imaging characteristicscomprise at least one among spectral response, gain, offset, distortion,and chromatic aberration.
 16. An array microscope for imaging an objectcomprising: a two-dimensional microscope array with a plurality ofmagnifying imaging systems disposed along a corresponding plurality ofoptical axes for imaging a picture of the object onto a detector, saidplurality of magnifying imaging systems being arranged in rows andconfigured to image respective sections of the object, wherein said rowsof imaging systems are staggered with respect to a linear direction ofscan across the object, such that each of the imaging systems acquiresimage data corresponding to a respective continuous strip of the objectalong said linear direction of scan; a scanning mechanism for producinga relative movement between the microscope array and the object, saidscanning mechanism operating continuously along said linear direction ofscan across the object so as to produce multiple adjacent image swathsof the object, each swath having a width defined by a field of view ofthe array microscope and a length defined by said scan across theobject; means for calibrating the array microscope to derive correctionfactors designed to correct imaging characteristics of said magnifyingimaging systems in the array microscope in order to normalize an outputthereof and produce images with uniform optical properties; means forapplying said correction factors to the multiple image swaths to obtainmultiple swaths of corrected images; and means for concatenating themultiple swaths of corrected images to produce a composite image of theobject; wherein said calibrating means consists of sample surfaces withpredetermined physical characteristics designed to produce target imageswith predetermined optical properties, so that deviations from saidpredetermined optical properties may be used to compute correctionfactors relative to said imaging characteristics.
 17. The arraymicroscope of claim 16, wherein said imaging characteristics comprise atleast one among spectral response, gain, offset, distortion, andchromatic aberration.